feat(core): add portable document ingestion contracts - #1178

Why

Hosted document ingestion needs one portable Basic Memory contract for generated Markdown, provenance, deterministic identity, and safe semantic parsing. Without that boundary, Core and Cloud would disagree about what a raw extraction note means, how retries converge, and which fields an enrichment agent may control.

This PR is the Core contract used by basic-memory-cloud#1650, implements the parser-neutral portion of #1006, and deliberately leaves historical revision indexing to #1177.

Specification

Scope

Core must provide strict, provider-neutral contracts for:

• a current generated Markdown note with type: document;
• a compact, queryable provenance note with type: document_ingestion_run;
• deterministic document and run-note identity derived from trusted inputs;
• canonical raw Markdown whose body is searchable but cannot mint observations or relations from arbitrary extracted syntax;
• a bounded structured enrichment output that cannot replace trusted source, extraction, ingestion, checksum, or identity fields.

Behavioral contract

1. source, extraction, ingestion, and document metadata is strict, nested, extra-forbidden, and serializable through either the YAML schema alias or the Python schema_ref field name.
2. Document and run identities are deterministic. Run identity includes source identity/checksum, extractor engine/version/profile/options, pipeline version, and prompt version.
3. Raw Markdown uses bm_parse_semantics: false; Core preserves body content for search while suppressing observation/relation parsing only for an explicit scalar false.
4. Run-note stage, output revision kind, output identity, and output path must agree with the trusted inputs.
5. Agent-provided body, tags, observations, and relations must round-trip through the real Markdown parser without hidden or additional semantics.
6. Enrichment starts from an exact raw canonical checksum and reconstructs the final trusted envelope server-side.
7. Public schema imports must not pull the heavy Markdown stack into lightweight CLI registration.

Non-goals

• selecting or running a PDF parser;
• Cloud queues, Tigris reads/writes, provider version IDs, or deployment policy;
• enabling production agent enrichment;
• historical revision storage, search, graph reconstruction, retention, or legal hold.

Acceptance Criteria

• document and document_ingestion_run have strict versioned Pydantic contracts.
• Nested provenance frontmatter is canonical, queryable, and round-trips through model serialization.
• Document, run, and sidecar paths are derived from stable trusted inputs.
• OCR page diagnostics reject coerced booleans, floats, and strings.
• Raw generated content remains body-searchable while arbitrary PDF syntax cannot create graph semantics.
• Run lifecycle state cannot contradict its raw/current output revision.
• Agent output cannot smuggle extra tags, observations, relations, context, or trusted provenance.
• Exact raw checksum compare-and-swap is required before enrichment assembly.
• Lightweight CLI imports remain lazy.
• The portable contract is exported for the Cloud implementation without adding a parser/runtime dependency.

What Changed

• Added the document, extraction, ingestion-run, revision, and agent-output models in src/basic_memory/schemas/document.py.
• Added deterministic UUID/path helpers, canonical Markdown parsing/assembly, SHA-256 helpers, lifecycle validation, and protected enrichment assembly.
• Exported the portable contracts from src/basic_memory/schemas/__init__.py.
• Added explicit document_ingestion note-object source support.
• Updated EntityParser to honor the raw semantic opt-out safely without dropping body content.
• Added focused schema, Markdown, runtime, and CLI-import regression coverage.

Implementation Details

• UUID5 run identity is recomputed from the source and pipeline fields instead of trusting a supplied UUID.
• Run-note output identity/path is recomputed from the source entity and source path.
• Canonical Markdown reconstruction protects system-owned metadata and validates the fully assembled semantic body, including fence/comment boundary cases.
• Agent tags use a canonical single-tag grammar; structured observations and relations must parse back to exactly the submitted objects.
• Markdown parser imports remain inside the validators that need them, preserving the lightweight CLI boundary.

Testing

Automated

• just fast-check: passed Ruff fix/check, formatting, and ty type checking.
• uv run pytest -q tests/schemas/test_document.py tests/markdown tests/test_runtime.py::TestRuntimeContracts::test_note_object_metadata_parses_safe_values_only: 128 passed; src/basic_memory/schemas/document.py reached 100% statement coverage.
• uv run pytest -q --no-cov tests/cli/test_cli_exit.py::test_bm_cli_import_does_not_load_heavy_stack: 1 passed.
• GitHub CI on 61d7b0dd55313076a081bad823eb69796d4b8426: unit, SQLite integration, Postgres integration, semantic, Milvus, static, CodeQL, DCO, and CLA jobs passed or were skipped as designed.
• git diff --check: passed.

Manual

• No UI or manual runtime verification applies to this parser-neutral contract PR.

Risks / Follow-ups

• Cloud must continue to validate only the trusted shapes it actually produces; additional future enrichment/history shapes can tighten their own invariants when implemented.
• Agent enrichment remains disabled until Cloud can prove exact raw Tigris materialization (basic-memory-cloud#1649).
• Current search and graph behavior remains head-only; the opt-in “BM data lake” is #1177.
• The commit is SSH-signed and DCO-signed-off. GitHub reports unknown_key, so this description does not claim GitHub Verified status.